Using pupil location to correct optical lens distortion

ABSTRACT

The disclosure relates generally to techniques for using information about a user&#39;s actual or predicted pupil location for correcting optical distortions that are specific to an optical lens and display assembly through which the user is viewing one or more images. The described techniques may include identifying and mapping optical distortions specific to an optical lens and display assembly, and using such mapped optical distortions to correct images displayed to a wearer or other user receiving images via the assembly, such as based at least in part on pupil location of the wearer or other user. As one example, the one or more optical lens may be mounted inside a head-mounted display (HMD) that also includes a display panel or other image source for an eye of a wearer, and if so one or more pupil tracking mechanisms may be integrated into the HMD.

TECHNICAL FIELD

The following disclosure relates generally to techniques for using pupillocation of a user to correct optical distortions from one or moreoptical lens being used to view a display panel or other image source,such as for use in a head-mounted display and/or in other devices inwhich one or more users receive images through one or more optical lens.

BACKGROUND

Demand for displays with heightened performance has increased with thedevelopment of smart phones, high-definition televisions, as well asother consumer electronic devices. The growing popularity of virtualreality and augmented reality systems, particularly those usinghead-mounted displays (“HMDs”), has further increased such demand.Virtual reality systems typically envelop a wearer's eyes completely andsubstitute a “virtual” reality for the physical reality in front of thewearer, while augmented reality systems typically provide asemi-transparent or transparent overlay of one or more screens in frontof a wearer's eyes such that a physical view is augmented withadditional information, and mediated reality systems may similarlypresent information to a viewer that combines real-world elements withvirtual elements. In many virtual reality and augmented reality systems,the movement of a wearer of such a head-mounted display may be trackedin various manners, such as via sensors in the head-mounted displayand/or external to it, in order to enable the images being shown toreflect user movements.

However, such head-mounted displays, with reduced distance between aviewer's eye and the display and often with a fully obscured field ofview, typically have complex performance requirements for optical lensin ways that are difficult to satisfy, let alone to do so atcost-effective levels, and other devices using displays with opticallens may have similar problems. In addition, manufacturing of suchhead-mounted displays can be difficult and costly, such as due tochallenges that include precise manufacturing tolerance requirements andlimitations in existing mass production capabilities. Accordingly, needsexist for improved techniques for using optical lens and formanufacturing head-mounted displays and other assemblies of one or moreoptical lens with additional components, including the need to correctfor distortions from the optical lens and to compensate forimperfections in such assemblies.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a top plan view of a head-mounted display systemwhich includes binocular display panels according to an exampleembodiment of the present disclosure.

FIGS. 2A-2G illustrate examples of use of optical lens, such as forhead-mounted displays, in particular manners in particular embodimentsin accordance with the described techniques.

FIG. 3 illustrates example computing devices and systems for performingat least some of the described techniques.

FIG. 4 illustrates an example embodiment of a Lens Distortion Mappingroutine.

FIG. 5 illustrates an example embodiment of a Pupil Location OpticalDistortion Correction routine.

DETAILED DESCRIPTION

The disclosure relates generally to techniques for using informationabout a user's eye location and/or movement as part of correctingoptical distortions that are caused by or otherwise specific to one ormore particular optical lens through which the user is viewing one ormore images. In at least some embodiments, pupil tracking techniques areused to determine the pupil location (e.g., in three dimensions) and/orpupil movement of a user who is using one or more optical lens in such amanner. As one example, the one or more optical lens may in someembodiments be mounted inside a head-mounted display (HMD) that alsoincludes a display panel or other image source for an eye of a wearer,and if so the pupil tracking mechanism may be integrated into the HMD,while in other embodiments, the described techniques may be used withother types of display devices, as discussed in greater detail below. Ifan HMD is in use and includes two separate optical lens that are eachused by one eye, each pupil may be tracked separately, in order toenable optical distortion corrections that are specific to the opticallens(es) through which that eye is receiving images—in at least someembodiments, the optical distortion corrections specific to an opticallens are further specific to a head-mounted display or other assemblyincluding the optical lens and one or more other elements (e.g., adisplay panel, a half-mirrored reflective element between a user's eyesview of the world, etc.), such as to reflect physical layout of theoptical lens and other elements and/or to reflect operation of the otherelements, and further discussion of optical distortion corrections areto be understood to include such assemblies and other elements in thoseembodiments. In addition, the use of the terms “lens” herein refers toany optical element that adjusts the path of light, includingreflective, refractive and other techniques, and a path of lightentering a user's eye may include various elements (e.g., display panel,lens, a half-mirrored reflective element between a user's eyes view ofthe world, etc.) in various orders and arrangements.

In at least some embodiments, the described techniques includeidentifying and mapping optical distortions specific to an optical lens,such as after the optical lens is mounted in an HMD or otherassembly/device having capabilities to display images that will beviewed by one or more users via the optical lens. Such opticaldistortions may be of various types, as discussed further below. Inaddition, the mapping of an optical lens' optical distortions mayinclude positioning an image sensor at each of multiple possiblelocations for a user's pupil, and identifying specific light-sensitivepositions within the image sensor that receive light when one or morecorresponding pixels within a display device are activated—such pixelsmay in some situations provide light of a specific color (e.g., red,green or blue), and are referred to at times as “sub-pixels” that may begrouped together into an aggregate pixel that provides white light whenthe constituent sub-pixels are all activated. After such pixel-to-pupilposition mapping information for a particular pupil location (alsoreferred to at times as a “static distortion mesh” or “volumetriccalibration field”) is generated, it may be stored in various manners,such as in one or more data structures for later use in adjusting animage to be displayed to a human eye's pupil at that pupil location.Such techniques for identifying and mapping optical distortions specificto an optical lens may further be performed at various times in variousembodiments, including at a time of manufacture of the device in whichthe optical lens is mounted or otherwise attached (e.g., by one or morecomputing systems operated by the manufacturer), and/or at a time ofinitial use of the device by a wearer or other user (e.g., by one ormore hardware processors that are part of or in communication with thedevice, such as one or more GPUs, or graphical processing units, and/orone or more CPUs, or central processing units). In other embodiments,the generation of such mapping information may be performed in part orin whole without using such a display panel and/or image sensor, such asby instead modeling the optical lens and simulating the optical effectsat different pupil locations as noted above. In yet other embodiments,some or all of the initial mapping information is generated in mannerdiscussed above using a display panel and image sensor, but modeling andsimulating may be used to modify such initial mapping information in oneor more manners (e.g., in a user-specific manner, such as to addressdistortions specific to an eye of the user and/or glasses or contactsworn by the user based on a prescription or other information thatprovides corresponding information). Additional details related to suchtechniques for identifying and mapping optical distortions specific toan optical lens are included below, and such techniques may be performedvia automated operations of a lens distortion mapping system in at leastsome embodiments, as discussed in greater detail below.

In at least some embodiments, the described techniques include usingmapped optical distortions specific to an optical lens to correct imagesdisplayed to a wearer or other user receiving images via the opticallens, such as based at least in part on pupil location of the wearer orother user. Such techniques for using mapped optical distortionsspecific to an optical lens to correct images displayed via the opticallens may include determining a wearer or other user's pupil location(e.g., via pupil tracking capabilities integrated into the devicecontaining the optical lens, or instead in an associated externaldevice) for use in displaying an image, such as an actual current pupillocation and/or a predicted future pupil location at a future time(e.g., at a defined number of future milliseconds) at which the imagewill be displayed. One or more of the predefined pupil locations forwhich mapping information is available may then be determined, such asto select one or more nearest predefined pupil locations (e.g., fourpredefined pupil locations surrounding the determined pupil location),and the mapping information for the selected predefined pupillocation(s) may then be used to adjust the image to be displayed, suchas to alter which pixels on the display device are illuminated so thatpositions within the determined pupil location receive lightcorresponding to the image before the adjustment, in order to correctthe optical distortions for the optical lens. Such techniques for usingmapped optical distortions specific to an optical lens to correct imagesdisplayed via the optical lens may be performed for various types ofdevices in which such optical lens are used, such as an HMD, camera,telescope, binoculars, etc., whether by one or more processors that areintegrated in such devices or instead are located in one or moreexternal computing systems that assist in display capabilities for thedevices. Additional details related to such techniques for using mappedoptical distortions specific to an optical lens to correct imagesdisplayed via the optical lens are included below, and such techniquesmay be performed via automated operations of a pupil location opticaldistortion correction system in at least some embodiments, as discussedin greater detail below.

For illustrative purposes, some embodiments are described below in whichvarious specific details are included for illustrative purposes, and insome cases some embodiments are simplified for the sake of brevity so asto not illustrate well-known components. For example, in someembodiments discussed below, particular types of display panels are usedin particular manners with particular types of optical lens (e.g., aspart of a head-mounted display for virtual reality and/or augmentedreality), including to use particular types of techniques as part ofcontrolling display operations for the display panel to reduce oreliminate optical distortion from the lens, such as based at least inpart on tracking pupil location and pupil movement of a user inparticular manners. However, it will be appreciated that the inventivetechniques may be used in a wide variety of other situations, includingwith other types of display devices and/or other types of determinationof a user's pupil location or other gaze direction, some of which arediscussed below.

FIG. 1 is a simplified top plan view of an HMD system 100 that includesa pair of near-to-eye display systems 102 and 104. The near-to-eyedisplay systems 102 and 104 include displays 106 and 108, respectively(e.g., OLED micro-displays), and respective optical lens systems 110 and112 that each have one or more optical lenses. The display systems 102and 104 may be mounted to frame 114 which includes a front portion 116,a left temple 118 and right temple 120. The two display systems 102 and104 may be secured to the frame 114 in an eye glasses arrangement whichcan be worn on the head 122 of a wearer user 124. The left temple 118and right temple 120 may rest over the user's ears 126 and 128,respectively, while a nose assembly (not shown) may rest over the user'snose 130. The frame 114 may be shaped and sized to position each of thetwo optical systems 110 and 112 in front of one of the user's eyes 132and 134, respectively. Although the frame 114 is shown in a simplifiedmanner similar to eyeglasses for explanatory purposes, it should beappreciated that in practice more sophisticated structures (e.g.,goggles, integrated headband, helmet, straps, etc.) may be used tosupport and position the displays systems 102 and 104 on the head 122 ofuser 124.

The HMD system 100 of FIG. 1 is capable of presenting a virtual realitydisplay to the user 124, such as via corresponding video presented at adisplay rate such as 30 frames (or images) per second or 90 frames persecond, while other embodiments of a similar system may present anaugmented reality display to the user 124. Each of the displays 106 and108 may generate light which is transmitted through and focused by therespective optical systems 110 and 112 onto the eyes 132 and 134,respectively, of the user 124. While not illustrated here, each of theeyes will typically include a pupil aperture through which light passesinto the eye, with a typical pupil size ranging from 2 mm (millimeters)in diameter in very bright conditions to as much as 8 mm in darkconditions, while the larger iris in which the pupil is contained mayhave a size of approximately 12 mm the pupil (and enclosing iris) maytypically move within the visible portion of the eye under open eyelidsby several millimeters in the horizontal and/or vertical directions,which will also move the pupil to different depths from the optical lensor other physical elements of the display for different horizontal andvertical positions as the eyeball swivels around its center (resultingin a three dimensional volume in which the pupil can move). The lightentering the user's pupils is seen by the user 124 as images and/orvideo. In some implementations, the distance between each of the opticalsystems 110 and 112 and the user's eyes 132 and 134 may be relativelyshort (e.g., less than 30 mm, less than 20 mm), which advantageouslycauses the HMD system 100 to appear lighter to the user since the weightof the optical systems and the display systems are relatively close tothe user's face, and also may provide the user with a greater field ofview. While not illustrated here, some embodiments of such an HMD mayinclude various additional internal and/or external sensors, such as toperform pupil tracking separately for each eye 132 and 134, to trackhead location and orientation (e.g., as part of head tracking), to trackvarious other types of movements and position of the user's body,cameras to record external images (e.g., of an environment), etc.

While the described techniques may be used in some embodiments with adisplay system similar to that illustrated in FIG. 1, in otherembodiments other types of display systems may be used, including with asingle optical lens and display device, or with multiple such opticallenses and display devices. Non-exclusive examples of other such devicesinclude cameras, telescopes, microscopes, binoculars, spotting scopes,surveying scopes, etc. In addition, the described techniques may be usedwith a wide variety of display panels or other display devices that emitlight to form images, which one or more users view through one or moreoptical lens. One non-limiting example of a display panel with whichsome embodiments of the described techniques may be used is discussed infurther detail in U.S. application Ser. No. 15/059,171, filed Mar. 2,2016 and entitled “Display With Stacked Emission And Control LogicLayers,” which is hereby incorporated in its entirety. In otherembodiments, the user may view one or more images through one or moreoptical lens that are produced in manners other than via a displaypanel, such as on a surface that reflects light from another lightsource in part or in whole.

As noted above, various types of optical distortions may be caused bydifferent types of lens and different types of optical effects, and maybe corrected via use of the described techniques. For example, FIGS.2A-2G illustrate examples of use of optical lenses, such as forhead-mounted displays, in particular manners in particular embodimentsin accordance with the described techniques. With respect to FIG. 2A, ahuman user's eye 205 is illustrated, with an iris section 210 thatincludes a pupil 215. In addition, FIG. 2A includes an example displaypanel 230 (shown from a side view), with an optical lens 220 (also shownfrom the side) situated between the eye 205 and the display panel 230.As the display panel 230 illustrates various light rays 225 from pixels(not shown) of the display panel 230, the light travels outward and intothe lens 220. In an ideal situation, the lens 220 bends the light rays225 from different portions of the display panel 230 so that therespective light rays emerging from the optical lens 220 are enteringthe pupil 215 at corresponding locations, so as to form the imagedisplayed on the display panel 230 on the pupil and inner portions ofthe eye. With respect to light entering a central axis 217 of theoptical lens 220 (with respect to both horizontal and vertical axes,although only the vertical axis is visible in this side view), and ifthe pupil 215 is similarly aligned with that central axis, the lens 220may perform little or no bending of the light rays emerging from thedisplay panel 230. It will be appreciated that the optical lens 220 anddisplay panel 230 are illustrated from the side, and light rays may beemitted from the display panel in not only a vertical manner as isillustrated but also in a horizontal manner (or depth with respect tothis side view) that is not illustrated in this example. Furthermore,while the light rays 225 are illustrated as leaving the display panel230 is purely a straight line for the purpose of this example, it willbe appreciated that some or all pixels of the display panel emit lightin multiple directions (or with at least some variation from beingpurely orthogonal to the surface of the display panel), even if focusedby per-pixel lenses (not shown).

FIG. 2B illustrates a further example of information that may bedisplayed on the display panel 230, which in the illustrated example isa test image with straight horizontal and vertical lines 232 and 233,respectively. A point 231 a corresponding to the central axis 217 isillustrated, as are other example points 232 a and 233 a that correspondto other points that are off the central axis. In addition to the imageshown for display panel 230, two alternative other resulting images 235b and 235 c are illustrated that show two types of possible opticaldistortions that may be caused on a viewer's pupil by light passingthrough one or more optical lenses. With respect to resulting image 235b, it illustrates an example of an optical distortion phenomenonreferred to as pincushion distortion, in which image magnificationincreases with the distance from the central optical axis 217, causing avisible effect in which lines are bowed inward, the farther from thecentral axis that they occur. Conversely, visual representation 235 cillustrates a different type of optical distortion referred to as barreldistortion, in which image magnification decreases with distance fromthe central optical axis, such that lines are increasingly bowedoutwards as they progress farther from the central optical axis. It willbe appreciated that such optical distortion effects may occur in anoptical lens even if it does not include errors or other distortionsthat are specific to that lens, such as based on the amount of curvatureof the lens relative to a distance on which the image is being focused.In addition, a particular lens may, in some cases, include combinationsof both pincushion and barrel distortion, sometimes referred to asmustache distortion, in which it begins as barrel distortion close tothe central optical axis and gradually turns into pincushion distortiontowards the periphery of the image.

FIG. 2C illustrates an example of another type of optical distortionphenomenon, referred to as chromatic aberration, which is caused in partby the different degrees of refraction of different color light rayswhen passing through a curved optical lens. In particular, it will beappreciated that a display panel 230 such as that illustrated in FIGS.2A-2C may include pixels of different colors, such as common use of red,green, and blue pixels in various combinations in an RGB display. Insituations in which a set of red, green and blue pixels are locatedalong the central optical axis 217, the light emitted from them (such asin this example being shown as 226 r, 226 g and 226 b for red, green andblue pixels, respectively) may pass through an optical lens along thatcentral axis with little or no bending of the different light rays, asreflected in the resulting light rays 236 r, 236 g and 236 b,respectively. Such light whose rays are parallel are referred to attimes as collimated light. Conversely, the farther the light rays arelocated from the central optical axis when they pass through the opticallens 220, with a greater degree of curvature of the optical lens atthose locations, the greater the variation in the amount of refractionbetween different light rays. Thus, with respect to example light rays227 r, 227 g and 227 b emitted from respective red, green and bluepixels at an area approximately one third of the distance from thecentral optical axis to the top of the display panel 230, the resultinglight rays 237 r, 237 g and 237 b are increasingly separated as theyleave the optical lens 220 and approach the eye 205. Similarly, forlight rays 228 r, 228 g and 228 b that are emitted near the top of thedisplay panel 230, the respective light rays 238 r, 238 g and 238 b thatare emitted from the optical lens 220 have even greater divergence,resulting in this example with two of the three light rays 238 not evenbeing received by the pupil 215. It will be appreciated that suchchromatic aberration effects may occur even if an optical lens does notinclude any errors or distortions specific to the lens, such as based onthe degree of curvature and the respective properties of differentcolors of light. In addition, FIG. 2C further illustrates an alternativepupil position 215′—as is shown, light from a single pixel (e.g., lightray 237 g) will enter the pupil at different angles as the pupil moveswithin the eyebox. Moreover, this effect may vary over the visual field,making certain objects or other elements being displayed appear toshift, stretch and/or compress as the pupil moves, thus seeming wobblyor “swimmy” and contributing to motion sickness if not corrected.Furthermore, while FIGS. 2B and 2C provide examples of some types ofoptical distortion phenomenon, various other types of opticaldistortions may occur in particular situations, as well as problemscaused by manufacturing defects, misalignments (e.g., between an opticallens and associated display panel), etc., and some or all of these maycorrected in whole or in part by the inventive techniques describedherein.

FIG. 2D continues the examples with respect to FIGS. 2A-2C, andillustrates a specific example of how an example optical lens 220 (shownas a single line without width for the purposes of illustration) maydistort an image being emitted by the display panel 230, which in thisexample again illustrates the same example test pattern with straighthorizontal and vertical lines 232 and 233, respectively, as previouslyillustrated in FIG. 2B. In this example, a portion 240 of an eye that isvisible under open eyelids (not shown) is illustrated, with an exampleiris 245 and pupil 250. In an ideal situation, a visual representationof the image from the display panel 230 will be projected onto the pupilvia the lens 220, including having the horizontal and vertical lines 232and 233 of the display panel 230 being shown in the same manner and inthe same relative locations within the pupil 250 as is illustrated.

However, due to errors specific to the particular lens 220, as well asother optical distortion effects as discussed previously and elsewhereherein (e.g., imperfect alignment and other manufacturing defects), theactual locations on the display panel that are projected to therespective portions of the pupil may not be perfectly aligned in theillustrated manner. For example, a center point 231 a of the displaypanel 230 in this example may correspond to an aggregate or combinationpixel with multiple sub-pixels (e.g., that each emits one of red, greenand blue light). However, even if the pupil is directed along thecentral axis 217 and is looking straight at the center point 231 a (suchthat a center of the pupil is aligned with the central axis),distortions in the optical lens 220 may still cause at least a slightshifting of light, such that the light that appears in the center point231 d within the pupil actually corresponds to and is emitted fromcombination pixel 241 to the right of the actual center point 231 a. Inthis example, combination pixel 241 is offset from the central location231 a in only the horizontal direction, but it will be appreciated thatsuch an offset may occur in the horizontal and/or vertical directions.In a similar manner, the display point 232 a in the upper right of thedisplay panel may be offset from the actual pixels in the display panelwhose light reaches that corresponding point 232 d within the pupil,such as, in this example, to have a combination pixel 242 that is offsetfrom the actual point 232 a in both the horizontal and verticaldirections. Similarly, with respect to point 233 a in the upper leftportion of the display panel, in this example, the optical distortionsof the lens 220 may cause different sub-pixels at different locations tocorrespond to a single point 233 d in the visual representation in thepupil, such as to have a red sub-pixel 243 r in one location, a bluesub-pixel 243 b in a different location, and a green sub-pixel 243 g ina third location, with those three different sub-pixels in effect actingas an aggregate combination pixel 243 with respect to the actual point233 d in the visual representation within the pupil. It will beappreciated that while a single red, green and blue sub-pixel is shownin this example corresponding to each of the viewpoints 231 a, 232 a and233 a, that various combinations of pixels in various locations maytogether provide light rays that converge on a particular point in avisual representation within a pupil in particular situations. Inaddition, while an optical lens may actually cause the visualrepresentation passing through the pupil to the retina to be flippedwith respect to the horizontal and/or vertical axes relative to theimage displayed on the display panel 230, the pupil's visualrepresentation is illustrated in these examples without such changes forthe sake of simplicity.

In order to address the optical distortions that are specific to thisparticular optical lens 220, the described techniques include performingoperations in at least some embodiments to map the various pixels of thedisplay panel 230 with respect to their actual effect through theoptical lens 220 on a representative pupil located at a position inwhich the actual human eye pupil will receive the light from such adisplay panel. In particular, in this example the display panel iscomposed of a large numbers of rows 252 and columns 253 of pixels, witheach such combination pixel in this example including red, green andblue sub-pixels. Thus, for example, with respect to example pixel 254,it includes sub-pixels 254 b (a blue sub-pixel), 254 g (a greensub-pixel) and 254 r (a red sub-pixel). In some such embodiments, themapping operation includes selecting combinations of one or moresub-pixels and successively illuminating each such group, anddetermining one or more corresponding positions within the pupil 250that receive light from that group of one or more pixels. For example,an image sensor (not shown) may be instead placed at the location atwhich the pupil would otherwise be located along the central axis 217,and particular light-sensitive positions within the image sensor (e.g.,positions from an array of such light-sensitive positions within theimage sensor) may be determined as receiving incoming light rays fromthat group of pixels. By progressively moving through some or all pixelsin the display panel, such as by illuminating (or activating) eachindividual sub-pixel or combination pixel separately in some suchembodiments, corresponding location points within the visualrepresentation of the pupil (as represented by the image sensor) may bedetermined for the illuminated pixels and that central axis pupillocation. The resulting information may then be used to map particularpixels to particular positions within the pupil for that central axispupil location and that optical lens, such as to provide the informationdiscussed above with respect to the actual effects of pixels 241, 242and 243 with respect to the corresponding locations 231 a, 232 a and 233a within the display panel.

After such a pixel-to-pupil position mapping is created for a displaypanel and a particular optical lens to be used with it, the resultingmapping information may be used to adjust the image that is actuallydisplayed on the display panel, in order to control how the resultingvisual representation on the pupil 250 occurs. For example, if an imagehas a white pixel at location 233 a of the display panel, the actualpixel(s) that are activated to have the corresponding image occur at thelocation 233 d of the pupil 250 may need to be changed to the aggregatecombination pixel 243 as discussed—thus, an automated processing routinemay alter the information for the display panel buffer (or the imageitself) so that the actual one or more pixels at location 233 a may beturned off, while the illustrated sub-pixels for aggregate combinationpixel 243 may be illuminated to cause that white point to be shown atlocation 233 d of the visual representation within the pupil. It will beappreciated that such a determination may be made for each pixel on thedisplay panel, in order to determine zero or more alternative actualpixels to illuminate to cause the original pixel to be shown at thecorrect location in the visual representation within the pupil. In thismanner, a human user that is using this optical lens 220 and displaypanel 230 may receive the displayed visual representation of straighthorizontal and vertical lines 232 and 233 on the visual representationin the pupil, even if the actual pixels illuminated on the display panel230 do not display such a representation in the absence of the opticallens.

It will be appreciated that if multiple optical lenses of a specifiedtype could be generated so that they are substantially identical andwithout any lens-specific errors or other distortions, such a predefinedmapping of one or more pupil locations to particular display panelpixels may be performed only a single time for a lens of that type, ifthe relative location of different such optical lenses may be specifiedwith sufficient accuracy relative to eye location and display panellocation.

FIG. 2E continues the examples discussed with respect to FIGS. 2A-2D,and in particular extends the technique discussed with respect to FIG.2D to situations in which the pupil moves within the visual portion 240of the eye away from the central optical axis of the lens. Inparticular, a pupil of a typical user may range within an areaillustrated as 255 in this example, and referred to at times as the“pupil box” (although the area 255 may have a shape other thanrectangular, as is shown). If the pupil moves away from the centraloptical axis and the resulting image displayed on the display panel 230is not adjusted, changes in various optical distortions may occur,including those previously discussed, as well as additional opticaldistortions that can occur if the pupil location movement is accompaniedby other physical movement of the user (e.g., the user's head) and theresulting images do not adapt quickly enough to pupil movement and otheruser movement.

In this example, the iris is not illustrated, with the example pupil 250of FIG. 2D again illustrated in the center of the eye portion 240(corresponding to the central axis 217), but with alternative pupillocations 250 a and 250 b also shown. For example, if a user moves theirpupil laterally to the left (shown here as moving to the right from thestandpoint of an observer looking at the person's eye), the pupil atlocation 250 a will now correspond at a location 218 in the optical lens220 that is significantly off the central axis 217 of the lens (notshown in FIG. 2B) in the horizontal direction. Accordingly, if theexample pixels 241, 242 and aggregate pixel 243 of FIG. 2D are againilluminated to correspond to the viewpoints 231 a, 232 a and 233 a, theoptical distortions caused by the lens 220 at the location 218 willcause a resulting image in the pupil at location 250 a that differssignificantly from that of the pupil 250 at its central axis location.As discussed in greater detail elsewhere herein, pupil tracking may beperformed in various manners. As one example, if the lens 220 anddisplay panel 230 are part of a head-mounted display, such as for one ofthe two eyes of a user wearing the HMD, the HMD may include internalcameras or other sensors to perform pupil location tracking for eacheye.

Instead, as one example of how such pupil movement may be handled, thedescribed techniques may be in some embodiments alter the actual pixelsthat are displayed in the display panel in a manner that corresponds tothe movement of the pupil location relative to its central axis location250. With respect to the example of the pupil at location 250 a, thedescribed techniques in this illustrated example may include performinga lateral horizontal translation of the pixels that are actuallyilluminated in the display panel, so as to provide, if possible, alateral translation of the image from the display panel so that the sameimage as would have occurred in FIG. 2D at the central axis location isnow displayed at the pupil location 250 a with the translated pixels, asillustrated with respect to lateral translation arrows 262 a, 263 a and261 a corresponding to view locations 233 a, 232 a and 231 a on thedisplay panel and corresponding positions 231 d, 232 d and 233 d (notshown) in the pupil. Thus, the described techniques may performcalculations to determine new pixels to display to illustrate thosepoints 233 a, 232 a and 231 a for the new pupil location 250 a, such asto perform in this example a lateral translation of each of thesub-pixels being used to a new location. For example, with respect topixel 241 that was previously used to display the point at view location231 a, a lateral translation of a defined amount may be used to select anew pixel 271 that will instead be displayed to cause the point at theview location 231 a to occur in the center of the pupil at the pupillocation 250 a, and new pixel 272 may similarly be used to represent theprevious pixel 242 to illuminate the viewpoint 232 a at the new pupillocation 250 a. It will be appreciated that different locations may havedifferent amounts of horizontal translation, such as in this example totranslate the pixel 242 near the right edge of the display only a smallamount, while the pixel 241 near the center of the display is translatedby a larger amount, and the combination aggregate pixel 243 near theleft edge of the image display is translated by an even greater amount,with the new pixels for the aggregate pixel to replace pixel 243 beingshown by sub-pixels 273 g, 273 b and 273 r. In addition, it will beappreciated that even if the pupil moves in only a lateral manner alonga horizontal axis as is shown from location 250 to 250 a, the resultingtranslation of pixels to use in the display panel may occur in a mannerother than purely horizontal as is illustrated in this example.

In addition, alternative pupil location 250 b indicates a differentexample in which the pupil location is moved a smaller amount, but inboth a horizontal and vertical direction to axis 219 within the opticallens, such as is reflected by example translation 261 b corresponding tothe central point 231 a being moved to the new pupil location 250 b.While corresponding pixels in the display panel are not illustrated inthis example for pupil location 250 b, it will be appreciated that asimilar translation may be performed to select new pixels for use inproviding the same visual representation at the new pupil location 250b, such as by interpolating and/or extrapolating the new position ofeach pixel based on an amount of movement of the pixel relative to atotal amount of possible movement (e.g., if the pupil location 250 acorresponded to 75% of the possible movement to the left within thepupil location area 255, each of the pixels in the display panel may bemoved 75% of the possible amount from that pixel to the edge of thedisplay panel in that horizontal direction). In addition, while onlysingle pixels continue to be used to represent a corresponding point inthe display panel, it will be appreciated that in some embodiments andsituations, multiple pixels at one or more locations may be used torepresent a single viewpoint in the display panel, depending on pupillocation and optical distortion specific to the particular lens 220 inuse (e.g., in combination with its assembled alignment with theparticular display 230).

FIG. 2F continues the examples of FIGS. 2A-2E, and illustrates furtheruse of the described techniques to manage situations in which a pupillocation of a user is mapped and then used for locations other than on acentral axis for an optical lens. In particular, in the example of FIG.2F, the same optical lens 220 is again illustrated, along with eyeportion 240 and pupil box movement area 255 in which the pupil may move.As discussed in greater detail with respect to FIG. 2D, in someembodiments the described techniques include generating a mapping fromdisplay panel pixel locations to corresponding points within a pupilthrough a specific optical lens 220 when the pupil is located along thecentral optical axis of the optical lens 220. FIG. 2F illustrates anextension of that technique in which such a corresponding mapping ofdisplay panel pixel locations to corresponding visual representationpoints within a pupil location is performed for each of a number ofdefined positions 270 within the three dimensional pupil box movementarea 255. In particular, in this example a grid 270 is illustrated of anumber of individual positions that each represents the center of apupil at that location, such as for position 270 c corresponding to apupil location of 250 c and optical axis location 216. For each suchposition in the grid 270, a mapping is made of the display panel pixellocations to corresponding positions within a pupil centered at thatgrid position 270, such as by moving an image sensor in three dimensionsfor each such corresponding three dimensional pupil position, as well asto address the specific structure of the head-mounted display or otherassembly that includes the optical lens. It will be appreciated that inother embodiments the multiple positions within the pupil box movementarea may be arranged in manners other than a grid, and that in someembodiments multiple pupil locations may be assessed and mapped at thesame time, such as by having different image sensors corresponding todifferent pupil locations simultaneously that each create a separatemapping for their respective pupil location given output from thedisplay panel pixels in the manner discussed with respect to FIG. 2D.

After such pixel-to-pupil location mappings are made for each of thepupil positions 270 for this specific optical lens 220, a pupil locationof a user may later be tracked during use of a device with that sameoptical lens 220 and display panel, in order to determine how to alterthe image being displayed on the display panel 230 to correspond to thespecific pupil location at which the user's eye is located. As notedelsewhere, various possible pupil tracking and eye tracking techniquesmay be used in different embodiments. For example, if a user's pupillocation is determined to be at location 250 c during such use, centeredaround defined pupil position 270 c, the previously generated mappingfor that defined pupil position may be used to determine how to alterthe image being displayed on the display panel. In this example, inorder to generate the test image of straight horizontal and verticallines on the display panel, particular groups of one or more pixels maybe chosen for each point on the display panel, in order to generate theimage in the correct location within the visual representation of thepupil if centered at defined pupil position 270 c. Furthermore, theparticular groups of pixels are chosen for each point on the display sothat any single point (e.g., any point on one of the lines) remains at astable angular relationship to the pupil, regardless of the position ofthe eye in the eye box, so that the angle of light as it enters thepupil for any specific point in the view is correct based on its virtualdistance to the eye (e.g., if something is far away, then parallax isminimal so the angular distance would not change noticeably, while thedescribed techniques are used in at least some embodiments to adjust forparallax for closer objects due to movement of the eye in the eye box—inother embodiments, the same techniques are used to correct problemsother than parallax, as discussed elsewhere herein). In addition, in atleast some embodiments, different mappings and/or adjustments are madefor different colors (e.g., for each of red, green and blue), such as toperform color and wavelength-specific corrections. For example, withrespect to point 231 a in the center of the display panel 230, pixel 281may be determined to provide a corresponding display at that location.In a similar manner, combination aggregate pixel 282 may be determinedto provide a visual representation of point 232 a at its respectiveposition in the pupil, which in this example includes differentsub-pixels 282 b, 282 g and 282 r at different locations within thedisplay panel. Similarly, an aggregate pixel 283 is determined toprovide the corresponding visual representation in the pupil for thedisplay point 233 a in this example. It will be appreciated that, whilethe pupil location is moved up both horizontally and vertically up andto the right (shown to the left in the image from the standpoint of theobserver), that the translation of corresponding pixels in the displaypanel may vary in different manners, such as based on specific opticaldistortions (including imperfections) in this particular optical lens220, such that the combination aggregate pixel 283 is generally to theleft of the actual point 233 a, while the new pixel 281 is generally tothe right of its actual point 231 a, and the combination aggregate pixel282 is generally below its corresponding view location 232 a. Moregenerally, when determining what value to assign to any single physicalpixel on the display, once the appropriate mapping is determined,sampling may be used in a source image for that pixel to includeappropriate amounts from one or more neighboring pixels. Furthermore,while the generation and use of mapping information for different threedimensional pupil locations is discussed above, in some embodiments suchcalibration may further be performed for other types of distortions,such as from dynamically adjusting lenses (e.g., Alvarez lenses orelectronically tunable lenses), which may have different distortioncharacteristics across the eyebox volume at different settings.

As previously noted, in some embodiments the described techniques mayfurther include performing not only a three dimensional pupil locationdetermination, but may further include performing activities to useinformation about one or more locations of the pupil to predict a futurelocation of the pupil, such as by using two or more previous pupillocations and interpolating or extrapolating where a future location maybe if movement continues in a similar manner. For example, if adetermination is being made of how to adjust the display panel 230 at acurrent time, but the actual display will not occur for some period offuture time (e.g., a specified number of milliseconds or seconds), suchtechniques may be performed to predict a location of the pupil at thatspecified future time, and to use that predicted future pupil locationfor use in adjusting the display panel image, so that the image that'sactually displayed at that future time corresponds to that actual futurepupil location (if the prediction is correct). Such pupil locationprediction in some embodiments and situations include a simple lineartranslation based on recent movement, while other embodiments may usemore detailed techniques, such as to use information about a specificuser and/or about a specific series of images being displayed to predictthat this particular user's eye location may change in a particularmanner over the next period of time until the specified future time,and/or to predict that any given user's eye location will move in aparticular manner based on particular images being displayed. Inaddition, in some embodiments, information may be obtained and used whendisplaying and adjusting an image about actual and/or predicted pupillocations at multiple times, including to combine that information invarious ways (e.g., to take an average of the multiple locations), asdiscussed elsewhere herein. Examples of techniques that may be used someembodiments for such pupil location prediction are described in U.S.application Ser. No. 15/258,551, filed Sep. 7, 2016 and entitled “SensorFusion Systems And Methods For Eye-Tracking Applications,” which isincorporated by reference in its entirety.

FIG. 2G continues the examples of FIGS. 2A-2F, and in particularillustrates additional manners in which predefined mappings of displaypanel pixels to particular pupil locations for a particular optical lens220 may be performed.

In the illustrated example of FIG. 2G, a central axis position 270 d isillustrated in the visual representation for the eye location 240, alongwith position 270 c as discussed with respect to FIG. 2F. As previouslydiscussed, if a pupil location (whether actual or predicted) correspondsto a particular position such as 270 c, the predetermined mapping forthat particular view pupil location 270 c may be used to determine theadjustments to be made to an image on the display panel in order toproduce the desired visual representation on the iris at that pupillocation.

Conversely, if a position of the pupil such as that as 270 e isdetermined or predicted, which is between multiple predefined pupilpositions 270 f-270 i but does not correspond exactly to a single one,various additional techniques may be used to adapt the correspondingimage for the display panel to that intermediate pupil location 270 e.For example, in some embodiments and situations, a nearest of the pupilpositions 270 may be selected and used to adjust the image, such as inthis example to use the mapping for pupil positions 270 f to correspondto an actual or predicted pupil location of 270 e. In other embodiments,two or more of the pupil positions 270 f-270 i may be selected and usedtogether to represent the actual or predicted pupil location of 270 e.For example, the information of the mappings for each of the fourpredefined positions 270 f-270 i may be combined together and aggregated(e.g., averaged) to create a new mapping corresponding to locationswithin those four predefined locations, such as by weighting the mappinginformation from the surrounding predefined positions based on theirdistance to the actual or predicted pupil location 270 e, or instead bycombining them all equally without weighting. Furthermore, when thepredefined pupil locations 270 are tracked in three dimensions, the oneor more pupil locations near an actual or predicted pupil position maysimilarly be measured in three dimensions, including in some embodimentsand situations to select multiple predefined pupil locations thatpartially or fully surround the actual or predicted pupil position.Thus, various techniques for interpolating or extrapolating from one ormore such predefined pupil locations to represent an actual or predictedpupil position, including in three dimensions, may be used in someembodiments.

Thus, in this manner, the pupil locations of a particular user may betracked and optionally predicted, and those pupil locations may be usedto perform optical lens-specific adjustments to an image being displayedin order to correct the optical distortions present in the optical lensand provide a corresponding visual representation at that pupil locationthat reduces or eliminates differences from that intended for the image.In addition, such techniques may be used for each image being displayed,such as for thirty frames per second by adapting each image in less thanthe time needed before the next frame is to be displayed, in order toprovide continuous video that is adapted to the changing pupil locationsof a user as he or she watches the respective images from the video andmoves his or her pupil.

The use of the described techniques may provide various benefits,including to reduce or eliminate the effects of some or all of the typesof optical distortion discussed herein. In particular, regardless of thepupil location, the use of the described techniques may provide the sameor substantially same visual representation of an image to the user,such as to provide a visual representation to a user's pupil with lightthat is substantially collimated even if the pupil location issignificantly off the central optical axis. In addition to thesebenefits, additional benefits in at least some such embodiments mayfurther one or more of the following: increasing the effectivefield-of-view available to the user, reducing or eliminating the needfor specialized optical lenses (e.g., Fresnel lenses), providing foruser-specific optical distortion corrections, providing correction forHMD fit, providing eye relief correction, allowing use of highlydistorting or non-collimating optics, etc.

FIG. 3 is a block diagram illustrating example computing devices andsystems for performing at least some of the described techniques. Inparticular, FIG. 3 includes one or more server computing devices 300that are suitable for performing at least some of the describedtechniques for identifying and mapping optical distortions specific toan optical lens, such as by executing an embodiment of a Lens DistortionMapping system 342 that operates on behalf of one or more clients (e.g.,a manufacturer of devices that include the optical lenses, a retailerthat sells devices having the optical lenses to end users, etc.). Inaddition, FIG. 3 also includes one or more end-user devices 350 that aresuitable for performing at least some of the described techniques forusing mapped optical distortions specific to an optical lens to correctimages displayed to a wearer or other user receiving images via theoptical lens, such as by executing an embodiment of a Pupil LocationOptical Distortion Correction system 365 that operates on behalf of oneor more clients (e.g., a retailer that sells the devices 350 to endusers, an end user operating a device 350, etc.). One or more optionalother computing systems 390 are also illustrated, with the variousdevices interconnected via one or more computer networks 385 (e.g., theInternet, one or more cellular telephone networks, etc.), including toenable communications between the computing systems, devices, and anyother systems or components implemented on them.

The example server computing device(s) 300 each includes one or morehardware processors 305 (e.g., one or more CPU processors, one or moreGPU processors, etc.), various input/output (“I/O”) components 310,storage 320, and memory 330. Illustrated I/O components in this exampleembodiment include a display 311, a network connection 312, acomputer-readable media drive 313, and other I/O devices 315 (e.g.,keyboards, mice or other pointing devices, microphones, speakers,etc.)—such I/O components may enable a variety of types of interactiontypes, including, for example, voice control, gesture control, etc. Theexample end-user devices 3350 are similarly illustrated as each havingone or more hardware processors 351 (e.g., one or more CPU processors,one or more GPU processors, etc.), one or more I/O components 352,memory 357, and storage 354. While some of the details illustrated withrespect to the server computing devices 300 are not illustrated withrespect to the devices 350 and other computing systems 390, the devices350 and system 390 may similarly include some or all of the same typesof components as the server computing devices 300. The end-user devices350 may further include additional components that are not illustratedwith respect to device(s) 300, such as one or more optical lenses 367and one or more other I/O devices 354 (e.g., one or more internal and/orexternal cameras, one or more speakers to provide sound to the ears ofthe wearer or other user, one or more pupil tracking systems, othertypes of movement sensors or other sensors, etc.). Similarly, if one ormore of the other computing systems 390 operates in conjunction with oneor more of the end-user devices 350, such as to provide motion trackingand/or image display capabilities, those other computing systems maysimilarly include additional components that are not illustrated withrespect to device(s) 300.

In this example, a Lens Distortion Mapping system 342 is executing inmemory 330 of the server computing device 300, along with one or moreoptional other programs 349. As discussed in greater detail elsewhereherein, the Lens Distortion Mapping system 342 may perform at least someof the described techniques for identifying and mapping opticaldistortions specific to an optical lens, such as with respect to one ormore of the end-user devices 350 (e.g., as part of the manufacturing ofthe devices 350 before they are provided to respective end users). Aspart of its operation, the system 342 may generate and/or use variousstored data, such as on storage 320, including data 321 about definedpupil location viewpoints for which to generate mapping data, data 323about lens and/or the devices in which they are mounted or otherwiseattached, and data 327 that results from performing the mappingoperations. The generated mapping data 327 may further be used orprovided to recipients in various manners, such as to store particularmapping data generated for one or more optical lenses of a particularend-user device 350 on that device for later use, such as on storage 356of the end-user device 350 as data 357. While the Lens DistortionMapping system 342 is implemented at least in part as a software systemin this example, such as with corresponding software instructions thatwhen executed program or otherwise configure the processor(s) 305 andthe server computing device(s) 300 to perform automated operations thatimplement at least some of the described techniques, it may beimplemented in other manners in other embodiments.

In addition, a Pupil Location Optical Distortion Correction system 365is executing on the end-user device 300, such as in part or in whole asa software program (not shown) in memory 362, and in part or in whole asspecialized hardware components (not shown) on the device 350. Thememory 362 may further optionally store one or more image displayprograms 363 that generate or otherwise provide images to be displayedon the end-user device (e.g., on one or more display panels 352), alongwith one or more optional other programs 364, although in otherembodiments an external system (e.g., one or more of the other computingsystems 390) may instead supply some or all of the images to the device350 to be displayed. As discussed in greater detail elsewhere herein,the Pupil Location Optical Distortion Correction system 365 may performat least some of the described techniques for using mapped opticaldistortions specific to an optical lens to correct images displayed to awearer or other user receiving images via the optical lens, such as forthe one or more optical lenses 367 (e.g., as part of displaying imagesto one or more end users). As part of its operation, the system 365 maygenerate and/or use various stored data, such as on storage 356,including data 357 that maps particular display panel pixels toparticular pupil location positions for one or more defined pupillocations, data 359 about pupil location tracking (e.g., as generated byone or more pupil tracking devices 353 or otherwise received from one ormore external systems), and optionally one or more images 358 to displayon the end-user device 350. The mapping data 357 may be received invarious manners, such as from Lens Distortion Mapping system 342 onserver computing device 300, although in other embodiments a singledevice or system (e.g., the end user device 350, server computing device300, other computing system 390, etc.) may execute embodiments of boththe Lens Distortion Mapping system and Pupil Location Optical DistortionCorrection system.

It will be appreciated that the illustrated computing systems anddevices are merely illustrative and are not intended to limit the scopeof the present invention. For example, computing device(s) 300 and/orend-user device(s) 350 may be connected to other devices that are notillustrated, including through one or more networks such as the Internetor via the Web. More generally, such a computing system or device maycomprise any combination of hardware that can interact and perform thedescribed types of functionality, such as when programmed or otherwiseconfigured with appropriate software, including without limitationdesktop computers, laptop computers, slate computers, tablet computersor other computers, smart phone computing devices and other cell phones,Internet appliances, PDAs and other electronic organizers, databaseservers, network storage devices and other network devices, wirelessphones, pagers, television-based systems (e.g., using set-top boxesand/or personal/digital video recorders and/or game consoles and/ormedia servers), and various other consumer products that includeappropriate inter-communication capabilities. For example, theillustrated systems 342 and/or 365 may include executable softwareinstructions and/or data structures in at least some embodiments, whichwhen loaded on and/or executed by particular computing systems ordevices may be used to program or otherwise configure those systems ordevices, such as to configure processors of those systems or devices.Alternatively, in other embodiments, some or all of the software systemsmay execute in memory on another device and communicate with theillustrated computing system/device via inter-computer communication. Inaddition, while various items are illustrated as being stored in memoryor on storage at various times (e.g., while being used), these items orportions of them can be transferred between memory and storage and/orbetween storage devices (e.g., at different locations) for purposes ofmemory management and/or data integrity.

Thus, in at least some embodiments, the illustrated systems aresoftware-based systems including software instructions that, whenexecuted by the processor(s) 305 and/or 355 and/or other processormeans, program the processor(s) to automatically perform the describedoperations for that system. Furthermore, in some embodiments, some orall of the systems may be implemented or provided in other manners, suchas at least partially in firmware and/or hardware means, including, butnot limited to, one or more application-specific integrated circuits(ASICs), standard integrated circuits, controllers (e.g., by executingappropriate instructions, and including microcontrollers and/or embeddedcontrollers), field-programmable gate arrays (FPGAs), complexprogrammable logic devices (CPLDs), etc. Some or all of the systems ordata structures may also be stored (e.g., as software instructionscontents or structured data contents) on a non-transitorycomputer-readable storage medium, such as a hard disk or flash drive orother non-volatile storage device, volatile or non-volatile memory(e.g., RAM), a network storage device, or a portable media article(e.g., a DVD disk, a CD disk, an optical disk, a flash memory device,etc.) to be read by an appropriate drive or via an appropriateconnection. The systems, modules and data structures may also in someembodiments be transmitted as generated data signals (e.g., as part of acarrier wave or other analog or digital propagated signal) on a varietyof computer-readable transmission mediums, including wireless-based andwired/cable-based mediums, and can take a variety of forms (e.g., aspart of a single or multiplexed analog signal, or as multiple discretedigital packets or frames). Such computer program products may also takeother forms in other embodiments. Accordingly, the present invention maybe practiced with other computer system configurations.

FIG. 4 is a flow diagram of an example embodiment of a Lens DistortionMapping routine 400. The routine may be performed by, for example, aLens Distortion Mapping system 342 of FIG. 3 and/or a system performingthe lens distortion mapping operations discussed with respect to FIGS.2D-2G and elsewhere herein, such as to generate information to correctthe optical distortions of a particular optical lens (e.g., in ahead-mounted display or other type of assembly) from multiple possiblepupil locations in advance of use of the optical lens (e.g., in thathead-mounted display or other type of assembly). While the illustratedexample of the routine is performed for a single lens at a single timeand for a single pupil location (referred to in the routine as a“viewpoint” of the pupil) at a time, it will be appreciated that such aroutine may be used in other manners, including to simultaneouslyperform determinations for multiple optical lenses and/or for multiplesuch viewpoints. In addition, it will be appreciated that theillustrated embodiment of the routine may be implemented in softwareand/or hardware as appropriate, and may be performed by, for example, asystem separate from an HMD or other device in which an optical lens ismounted, such as before use of the device by a user begins, although inother embodiments the actual HMD or other device may instead performsome or all of the mapping techniques, such as at the beginning of usefor any users or for a specific user.

The illustrated embodiment of the routine begins at block 405, whereinan indication is received of an optical lens to map, with the opticallens optionally being mounted in a head-mounted display or other housingwith a display panel (or other display device) to be used along withthat optical lens. The routine then continues to block 415 to receiveinformation about a group of one or more pupil location viewpoints tomap, such as a single central optical axis pupil location viewpoint, orinstead multiple different possible pupil location viewpoints within apupil movement box area (with the central optical axis pupil locationviewpoint optionally being one of the viewpoints).

The routine then continues to perform blocks 425-465 for each such pupillocation viewpoint, in order to generate a mapping data structure forthe optical lens and that pupil location viewpoint that includesinformation for mapping the display panel pixels to correspondingpositions within an example pupil (represented in this example by animage sensor) at that pupil location viewpoint. In particular, theroutine continues after block 415 to block 425 to select the nextviewpoint to map, beginning with the first. In block 435, the routinethen positions an image sensor, if not already so positioned, at thatselected viewpoint location, with the image sensor representing thehuman pupil and having an array of light-sensitive points. After block435, the routine continues to block 445 to successively activate groupsof one or more display panel pixels on the display panel that ispositioned opposite the lens relative to the image sensor, and to mapone or more corresponding light-sensitive point positions on the imagesensor that receive light from the activated one or more pixels. Theprocedure in block 445 continues until all pixels on the display panelare activated, although in other embodiments and situations only subsetsof the display panel pixels may be activated (e.g., representativepixels at different locations, for only a subset of the display panel,etc.). After block 445, the routine continues to block 455 to determine,for each light-sensitive point position on the image sensor, one or moredisplay panel pixels from which light was received at that image sensorpoint, such that activating those pixels will cause that position on theimage sensor (or later, a pupil) to receive corresponding light. Afterthe determination is made, the routine generates a correspondingpixel-to-pupil location position mapping data structure for the selectedviewpoint in block 455, and in block 465 stores the generated mappingdata structure for the viewpoint and the optical lens.

After block 465, the routine continues to block 475 to determine ifthere are more viewpoints to map, and if so returns to block 425 toselect a next viewpoint. Otherwise, the routine continues to block 485to optionally aggregate the data for multiple viewpoints that weremapped into an overall information data structure for use with theoptical lens, store the data for the optical lens, and provide it to oneor more requesters if appropriate (e.g., a requester who provided theinstructions with respect to block 405).

After block 485, the routine continues to block 495 to determine whetherto continue, such as until an explicit indication to terminate isreceived. If it is determined to continue, the routine returns to block405 and waits for information about a next optical lens to map, andotherwise continues to block 499 and ends.

FIG. 5 is a flow diagram of an example embodiment of a Pupil LocationOptical Distortion Correction routine 500. The routine may be performedby, for example, a Pupil Location Optical Distortion Correction system365 of FIG. 3 and/or corresponding systems described with respect toFIGS. 2E-2G, such as to adjust one or more images to be displayed on adisplay panel (or other display device) in a manner specific to aparticular optical lens and pupil location, and optionally to furtheradjust such information specific to a particular user. While theillustrated embodiment of the routine is displayed with respect to aparticular image at a time, such as for a single image (whether byitself or as a part of a series of related images), it will beappreciated that other embodiments of the routine may be performed inother manners, such as to simultaneously adjust multiple images (e.g.,to perform adjustments for multiple images to be displayed in rapidsuccession if the pupil location of the user is not expected to differsignificantly between the display of those multiple images; to performadjustments for two or more images to be simultaneously displayed viatwo or more display panels and associated optical lenses, such as fortwo optical lenses in an HMD; etc.). In addition, it will be appreciatedthat the illustrated embodiment of the routine may be performed by, forexample, a particular HMD or other device in which the optical lens isincluded, such as dynamically while the images are received andpresented, and may be implemented in software and/or hardware asappropriate.

The example embodiment of the routine begins at block 505, where anindication is received of an optical lens, and in which the routineobtains pixel-to-pupil location position mapping information for one ormore pupil location viewpoints for the lens, such as informationpreviously generated in FIG. 4 for that optical lens. After block 505,the routine continues to block 515, where it optionally obtainsinformation about a specific user and any user-specific distortioncorrections to apply or images displayed to that user (e.g., based on ananalysis of the user-specific distortion corrections performed by theHMD or other device; based on information received from an externalsource, such as an optical exam; etc.), although in some embodimentssuch user-specific optical distortion corrections may not be used. Inthe illustrated example of the routine, the routine then continues toblock 525 to initiate pupil tracking for the optical lens and the user,although in other embodiments one or more separate systems may be usedto perform such pupil tracking, and the routine may instead receivepupil tracking information from those other systems.

The routine then continues to perform blocks 545-585 for each of one ormore images to be displayed to the user through the optical lens, suchas a series of image frames in a video, or instead a single stand-aloneimage. In particular, in block 545 the routine tracks the user's pupillocation, such as periodically or continuously, such as for use with thenext image to be displayed. As previously discussed, in some embodimentsthe pupil location may be determined using not only an actual currentlocation but a predicted future location at a specified future time,such as to correspond to an amount of time before an adjusted image willactually be displayed to the user. In block 555, the routine thendetermines one or more of the nearest defined pupil location viewpointsfor which the mapping information was generated that corresponds to thedetermined pupil location, and to retrieve the corresponding mappinginformation for those one or more nearest pupil viewpoints to use. Sucha determination of one or more nearest defined pupil location viewpointsmay be performed in various manners, such as by using a predefined ordynamically defined distance threshold (e.g., the distance to thenearest pupil location viewpoint, to the nearest four pupil locationviewpoints, etc.). In addition, while not illustrated in this exampleroutine, in some embodiments the routine may further obtain and useinformation about actual and/or predicted pupil locations at multipletimes, and combine the information to determine a pupil position to use(e.g., to take an average of the multiple locations). For example, theactual and/or predicted pupil location may be determined at some timebefore a corresponding adjusted image will be displayed to the user, andthat information may be combined with information about actual and/orpredicted pupil location for other times such as at one or more previoustimes of displaying one or more previous images, at a later time ofactually initiating the adjusting of the current image, etc.

The routine then obtains the next image to be displayed in block 560. Inblock 565, the routine then determines whether to use a single definedviewpoint or multiple defined viewpoints to adjust the image, althoughother embodiments of the routine may always use only a single viewpointor always use multiple viewpoints. In the illustrated example of theroutine, if it is determined to use a single defined viewpoint, such asif the determined pupil location is associated with a particular definedviewpoint (e.g., within a minimum threshold distance), the routinecontinues to block 575 to use the mapping information for that viewpointto determine how to alter the pixel information to be displayed for theimage, as well as to further optionally alter the image pixelinformation based on any user-specific information, such that an alteredor modified set of pixels to illuminate is determined so that thedesired image will be displayed at the determined pupil location giventhe optical lens distortions and any user-specific optical distortions.

If it is instead determined in block 565 to use multiple definedviewpoints, the routine continues instead to block 575 where two or moresuch defined viewpoints are selected to be used, and in which themapping information for those multiple selected viewpoints is combined,optionally weighting the mapping information for the differentviewpoints (e.g., based on the difference in location of the determinedpupil location from those viewpoints, in one or more other definedmanners, etc.). For example, if two selected viewpoints indicate twodifferent pixels to illuminate for a particular pupil location position,both of those pixels may be illuminated, or instead one or more otherpixels may be determined (e.g., by selecting one or more such otherpixels between those two pixels). After block 575, the routine continuesto block 580 to alter the pixel information for the image to bedisplayed based on the combined mapping information and optionally anyuser-specific information, in a manner similar to that discussed withrespect to block 570.

After blocks 570 or 580, the routine continues to block 585 to outputthe altered image pixel information for display, whether directly to thedisplay panel, or instead to another system or routine that performs theactual image display. The routine then continues to block 587 todetermine whether there are more images to be altered in the describedmanner, and if so returns to block 535 to wait for the next image to bedisplayed. Otherwise, the routine continues to block 595 to determinewhether to continue, such as until an explicit indication to terminateis received. If it is determined to continue, the routine returns toblock 505 and waits for an indication of another lens for which suchimage modifications are to be performed, and otherwise continues toblock 599 and ends.

It will be appreciated that in some embodiments the functionalityprovided by the routines discussed above may be provided in alternativeways, such as being split among more routines or consolidated into fewerroutines. Similarly, in some embodiments illustrated routines mayprovide more or less functionality than is described, such as when otherillustrated routines instead lack or include such functionalityrespectively, or when the amount of functionality that is provided isaltered. In addition, while various operations may be illustrated asbeing performed in a particular manner (e.g., in serial or in parallel)and/or in a particular order, those skilled in the art will appreciatethat in other embodiments the operations may be performed in otherorders and in other manners. It will similarly be appreciated that thedata structures discussed above may be structured in different manners,including for databases or user interface screens/pages or other typesof data structures, such as by having a single data structure split intomultiple data structures or by having multiple data structuresconsolidated into a single data structure. Similarly, in someembodiments illustrated data structures may store more or lessinformation than is described, such as when other illustrated datastructures instead lack or include such information respectively, orwhen the amount or types of information that is stored is altered.

From the foregoing it will be appreciated that, although specificembodiments have been described herein for purposes of illustration,various modifications may be made without deviating from the spirit andscope of the invention. In addition, while certain aspects of theinvention are presented at times in certain claim forms, or may not beembodied in any claims at some times, the inventors contemplate thevarious aspects of the invention in any available claim form. Forexample, while only some aspects of the invention may be recited at aparticular time as being embodied in a computer-readable medium, otheraspects may likewise be so embodied.

What is claimed is:
 1. A method comprising: attaching, to a head-mounteddisplay device that includes a display panel, an optical lens betweenthe display panel and an eye position area designed for a human eye toreceive images generated by the display panel; generating, by one ormore computing systems, and for each of a plurality of pupil locationsat the eye position area from which an image from the display panel maybe received through the optical lens, optical distortion mappinginformation at the pupil location from the optical lens attached to thehead-mounted display device, wherein the generating includes, for eachof the plurality of pupil locations; positioning an image sensor havingan array of multiple light-sensitive positions at the pupil location;for each of a plurality of different groups of pixels, successivelyactivating the group of pixels on the display panel and determiningresulting light-sensitive positions of the image sensor that receivelight from the activated group of pixels through the optical lens; andstoring an optical distortion mapping data structure for the opticallens and the pupil location that maps light-sensitive positions at thepupil location to corresponding pixels on the display panel; and using,by at least one hardware processor associated with the head-mounteddisplay, the generated optical distortion mapping information to displayimages through the optical lens to a human eye at the eye position area,including, for each of the images: determining an actual pupil locationof a pupil of the human eye using pupil tracking capabilities of thehead-mounted display; selecting one or more of the pupil locations ofthe plurality that are within a defined distance of the determinedactual pupil location; using the generated optical distortion mappinginformation in one or more stored optical distortion mapping datastructures for the selected one or more pupil locations as part ofadjusting the image to reduce optical distortions caused by the opticallens at the determined actual pupil location, including to changelocations of pixels to be activated for the adjusted image to causelight received at the determined pupil location to correspond to theimage before adjustment; and displaying the adjusted image on thedisplay panel.
 2. The method of claim 1 wherein the head-mounted displaydevice further includes a second optical lens between a second displaypanel and a second eye position area designed for a second human eye toreceive images generated by the second display panel, and wherein themethod further comprises: generating, by one or more computing systems,and for each of a plurality of second pupil locations at the second eyeposition area from which an image from the second display panel may bereceived through the second optical lens, second optical distortionmapping information for the second optical lens at the second pupillocation, wherein the second optical distortion mapping information isspecific to the second optical lens and is separate from the opticaldistortion mapping information for the optical lens; and using, by atleast one hardware processor associated with the head-mounted display,the generated second optical distortion mapping information to displaysecond images through the second optical lens to a second human eye atthe second eye position area, including, for each of the second images:determining a second actual pupil location of a second pupil of thesecond human eye using the pupil tracking capabilities of thehead-mounted display; selecting one or more of the second pupillocations that are within the defined distance of the determined secondactual pupil location; using the generated second optical distortionmapping information for the selected one or more second pupil locationsas part of adjusting the second image to reduce optical distortionscaused by the second optical lens at the determined second actual pupillocation; and displaying the adjusted second image on the second displaypanel.
 3. The method of claim 2 wherein the displayed images and thedisplayed second images are frames of recorded or generated videoinformation, and wherein the using of the generated optical distortionmapping information to display the images and the using of the generatedsecond optical distortion mapping information to display the secondimages is performed in a real-time manner to display the videoinformation at a defined rate that includes multiple frames per second.4. A computer-implemented method comprising: generating, by one or morehardware processors, and for an optical lens positioned between adisplay device and an eye position area having a plurality of pupillocations at which an image from the display device may be receivedthrough the optical lens, optical distortion mapping information foreach of the plurality of pupil locations, wherein the generatingincludes, for each of the plurality of pupil locations, successivelyactivating different groups of pixels on the display device anddetermining resulting positions at the pupil location that receive lightfrom the activated groups of pixels through the optical lens; and using,by at least one hardware processor, the generated optical distortionmapping information to display one or more images through the opticallens to a human eye, including, for each of the one or more images:determining a pupil location of a pupil of the human eye; selecting oneor more pupil locations of the plurality that are within a defineddistance of the determined pupil location; using the generated opticaldistortion mapping information for the selected pupil locations as partof adjusting the image to reduce optical distortions caused by theoptical lens, including to change locations of pixels to be activated onthe display device for the adjusted image to cause light received at thedetermined pupil location to correspond to the image before adjustment;and initiating display of the adjusted image on the display device. 5.The computer-implemented method of claim 4 wherein the using of thegenerated optical distortion mapping information for one of the one ormore images includes performing the selecting of the one or more pupillocations for the one image by identifying one pupil location of theplurality that is closest to the determined pupil location for the oneimage, approximating optical distortion mapping information for thedetermined pupil location for the one image by using a distance betweenthe determined pupil location and the identified one location to modifythe optical distortion mapping information for the identified one pupillocation, and using the approximated optical distortion mappinginformation for the adjusting of the one image.
 6. Thecomputer-implemented method of claim 5 wherein the one or more imagesinclude multiple images, wherein the using of the generated opticaldistortion mapping information for a second image of the multiple imagesincludes performing the selecting of the one or more pupil locations forthe second image by identifying multiple pupil locations of theplurality that are within the defined distance to the determined pupillocation for the second image, approximating optical distortion mappinginformation for the determined pupil location for the second image bycombining the optical distortion mapping information for the identifiedmultiple pupil locations, and using the approximated optical distortionmapping information for the adjusting of the second image.
 7. Thecomputer-implemented method of claim 5 wherein the adjusting of the oneimage includes, for each of at least some pixels of a plurality ofpixels to activate for the one image, determining one or more otherpixels of the plurality to activate to cause light to be displayed at aposition in the determined pupil location corresponding to the pixel. 8.The computer-implemented method of claim 4 wherein the using of thegenerated optical distortion mapping information for one of the one ormore images includes performing the selecting of the one or more pupillocations for the one image by identifying multiple pupil locations ofthe plurality that are within the defined distance to the determinedpupil location for the one image and surround the determined pupillocation at least in part, approximating optical distortion mappinginformation for the determined pupil location for the one image by usingcombining the optical distortion mapping information for the identifiedmultiple pupil locations, and using the approximated optical distortionmapping information for the adjusting of the one image.
 9. Thecomputer-implemented method of claim 8 wherein the adjusting of the oneimage includes, for each of at least some pixels of a plurality ofpixels to activate for the one image, determining one or more otherpixels of the plurality to activate to cause light to be displayed at aposition in the determined pupil location corresponding to the pixel.10. The computer-implemented method of claim 4 wherein the adjusting ofan image to reduce optical distortions caused by the optical lensfurther includes obtaining information about additional opticaldistortions specific to the human eye to which the image is to bedisplayed, and further adjusting the image to correct the additionaloptical distortions.
 11. The computer-implemented method of claim 4wherein the activated pixels are each a subpixel configured to, inoperation, emit light of one of red, green or blue from use of differentemissive materials or use of different color filters.
 12. Thecomputer-implemented method of claim 4 wherein the determining of thepupil location of the pupil includes using pupil tracking capabilitiesto determine a current actual pupil location, and adjusting thedetermined current actual pupil location to reflect a predicted futurelocation of the pupil at a future time at which the display of theadjusted image on the display device is to occur.
 13. Thecomputer-implemented method of claim 4 wherein the optical lens and thedisplay device are part of a head-mounted display, wherein the human eyeis an eye of a human wearing the head-mounted display, and wherein theat least one hardware processor includes a graphics processing unit ofthe head-mounted display.
 14. The computer-implemented method of claim13 wherein the one or more processors are part of one or more computingsystems involved in manufacturing of the head-mounted display.
 15. Thecomputer-implemented method of claim 4 wherein the optical lens and thedisplay device are part of one or more of a camera, a telescope, amicroscope, a surveying scope or binoculars.
 16. Thecomputer-implemented method of claim 4 wherein the display deviceincludes a plurality of pixels, and wherein the generating of theoptical distortion mapping information for the optical lens and for eachof the plurality of pupil locations includes: positioning an imagesensor having multiple light-sensitive positions at the pupil location;successively activating the different groups of one or more pixels onthe display device and determining resulting light-sensitive positionsof the image sensor that receive light from the activated groups ofpixels through the optical lens, wherein the one or more pixels in afirst group comprises pixels that emit only a first color, and the oneor more pixels in a second group comprises pixels that emit only asecond color different from the first color; and storing an opticaldistortion mapping data structure for the optical lens and the pupillocation that maps light-sensitive positions at the pupil location tocorresponding pixels on the display device.
 17. The computer-implementedmethod of claim 16 wherein each of the plurality of pupil locationsincludes a plurality of positions within the pupil location, and whereinthe generating of the optical distortion mapping information for theoptical lens and for each of the plurality of pupil locations furtherincludes determining, for each of the plurality of positions within thepupil location, one or more pixels of the plurality of pixels thatprovide light to the position for the pupil location.
 18. Anon-transitory computer-readable medium having stored contents thatcause at least one hardware processor to perform activities that includeat least: obtaining, for an optical lens positioned between a displaydevice and an eye position area having a plurality of pupil locations atwhich an image from the display device may be received through theoptical lens, optical distortion mapping information for each ofmultiple pupil locations of the plurality, wherein the obtaining of theoptical distortion mapping information includes generating, for each ofthe multiple pupil locations, the optical distortion mapping informationfor that pupil location by successively activating different groups ofpixels on the display device and determining resulting positions at thatpupil location that receive light from the activated pixels through theoptical lens, and storing a resulting optical distortion mapping datastructure for the optical lens and that pupil location; and using, bythe at least one hardware processor, the optical distortion mappinginformation to display one or more images through the optical lens to ahuman eye, including, for each of the one or more images: determining apupil location of a pupil of the human eye; determining a distance fromthe determined pupil location to at least one pupil location of themultiple pupil locations; using the optical distortion mappinginformation and the determined distance as part of adjusting the imageto reduce optical distortions caused by the optical lens, including tochange locations of pixels to be activated on the display device for theadjusted image to cause light received at the determined pupil locationto correspond to the image before adjustment; and initiating display ofthe adjusted image on the display device.
 19. The non-transitorycomputer-readable medium of claim 18 wherein the one or more pupillocations of the plurality include multiple pupil locations, wherein theobtaining of the optical distortion mapping information includesmodeling optical characteristics of the optical lens, and generating,for each of the multiple pupil locations and based at least in part onthe modeled optical characteristics, the optical distortion mappinginformation for that pupil location by simulating activation of pixelson the display device and determining resulting positions at that pupillocation that receive light from the simulated activated pixels throughthe optical lens, and storing a resulting optical distortion mappingdata structure for the optical lens and that pupil location, and whereinthe performing of the activities further includes selecting the at leastone pupil location of the multiple locations to use the adjusting basedat least in part on the determining distance.
 20. The non-transitorycomputer-readable medium of claim 18 wherein the determining of thepupil location of the pupil includes using pupil tracking capabilitiesto determine a current actual pupil location, and adjusting thedetermined current actual pupil location to reflect a first predictedfuture location of the pupil at a future time at which the display ofthe adjusted image on the display device is to occur.
 21. Thenon-transitory computer-readable medium of claim 20 wherein thedetermining of the pupil location occurs at a first time, wherein theusing of the optical distortion mapping information occurs at a secondtime and includes predicting at the second time a second future locationof the pupil for the future time at which the display of the adjustedimage on the display device is to occur, and wherein the adjusting ofthe image to reduce optical distortions caused by the optical lensincludes combining information about the first predicted future locationand the second predicted future location.
 22. The non-transitorycomputer-readable medium of claim 18 wherein the at least one hardwareprocessor includes at least one graphics processing unit in ahead-mounted display that includes the display device and the opticallens and the eye position area and a pupil tracking system, and whereinthe determining of the pupil location includes using the pupil trackingsystem as part of the determining.
 23. The non-transitorycomputer-readable medium of claim 18 wherein the computer-readablemedium is a memory of a computer system that includes the at least onehardware processor, and wherein the stored contents are instructionsthat, when executed, program the computer system to perform theactivities.
 24. A system, comprising: one or more hardware processors;and a pupil location optical distortion correction system that isconfigured to cause at least one of the one or more hardware processorsto perform activities including: obtaining, for an optical lenspositioned between a display device and an eye position area having aplurality of pupil locations at which an image from the display devicemay be received through the optical lens, optical distortion mappinginformation for each of multiple pupil locations of the plurality,wherein the obtaining of the optical distortion mapping informationincludes generating, for each of the multiple pupil locations, theoptical distortion mapping information for that pupil location bysuccessively activating different groups of pixels on the display deviceand determining resulting positions at that pupil location that receivelight from the activated pixels through the optical lens, and storing aresulting optical distortion mapping data structure for the optical lensand that pupil location; and using the optical distortion mappinginformation to display one or more images through the optical lens to ahuman eye, including, for each of the one or more images: obtaininginformation about a pupil location of a pupil of the human eye;selecting one or more pupil locations of the multiple pupil locationsbased at least in part on respective distances between the selected oneor more pupil locations and the determined pupil location; using theoptical distortion mapping information for the selected pupil locationsas part of adjusting the image to reduce optical distortions caused bythe optical lens, including to change locations of pixels to beactivated on the display device for the adjusted image to cause lightreceived at the determined pupil location to correspond to the imagebefore adjustment; and initiating display of the adjusted image on thedisplay device.
 25. The system of claim 24 further comprising a lensdistortion mapping system that is configured to cause at least one ofthe one or more hardware processors to generate the obtained opticaldistortion mapping information for each of the plurality of pupillocations by, for each of the multiple pupil locations, activatingpixels on the display device and determining resulting positions at thepupil location that receive light from the activated pixels through theoptical lens, and storing a resulting optical distortion mapping datastructure for the optical lens and the pupil location.
 26. The system ofclaim 24 further comprising a pupil tracking system, and wherein theobtaining of the information about the pupil location includes using thepupil tracking system to determine the pupil location.
 27. The system ofclaim 24 further comprising a head-mounted display that includes thedisplay device and the optical lens and the eye position area, andwherein the one or more hardware processors include at least onegraphics processing unit in the head-mounted display.
 28. The system ofclaim 24 wherein the pupil location optical distortion correction systemincludes software instructions for execution by at least one of the oneor more hardware processors.